This invention relates to a network of nonlinear devices having continuous (graded response) input-output relations for parallel processing of input signals, and obtaining useful collective decisional responses (to which all of the input signals make a contribution to some degree in the range from 0 to 100%), such as in retrieving discrete information from the network functioning as a content addressable (associative) memory, or in finding a discrete solution to a problem. Because of similarities with real biological neurons, the nonlinear devices are sometimes referred to hereinafter as "neurons" or "neuronal circuits," and the parallel processing of input signals for a collective decision as "neuronal computation."
Many practical problems take such an enormous amount of computation that a solution in real time is not possible. An example is selecting from memory information that best satisfies known characteristics or descriptors, which may be referred to as "clues," but which are insufficient to completely define the information thus retrieved. The present device differs from associative memories of the prior art which use a binary word as a search key that represents a particular value of all or some of the bits in the information word. The key is then compared in a discrete or logical calculation with corresponding tag bits of all stored words, similar to retrieving a particular automobile part in inventory where a code affixed to the part is used to retrieve the part itself. Such an associative memory will access only a particular word of information, unless the key is loose, i.e., has too few attributes that define properties of the information desired. There is no simple representation possible of the degree of confidence to be placed in particular elements of the key. One is then faced (in software or hardware) with the task of reviewing a long list of information words that match the loose key. It would be desirable to have an associative memory that directly and rapidly accesses only one particular information word that best matches the clues, to the exclusion of all others. If the user then determines that was not the correct information, he need only revise the clues.
Pattern recognition is another example of such a problem. In order for reasonable scale special purpose digital computers to cope with recognition tasks in real time, the sensory data must be in a particular format for recognition of a limited set of objects that must be presented in a predetermined orientation. An optical character reader, for example, is limited to one or a few sets of fonts, and each character to be recognized must be properly oriented. While some defects in the characters may be tolerated, major defects or change in font cannot. This is because of the nature of the logic used in a digital system for reducing the input data to the desired recognition decision. Each binary logic gate in a digital system will typically obtain input signals from two or three other gates, and a huge member of independent binary decisions are made in the course of a computation. The gates are influenced at any stage only by process data in a small amount of information. Intermediate solution signals must be stored for use in the next phase of the solution, and many successive phases must be used in order to evaluate the information as a whole. In the present invention, each nonlinear device gets inputs from many or all other devices for a collective decision to be made (in a single phase or convergence time) from all of the input signals presented on the basis of the simultaneous interactions of the devices in the network, which may be expanded virtually without limit to accommodate as large a base of input information as may reasonably be expected for the nature of the problem.
In order that the nature and full scope of the invention be better appreciated, an associative memory will first be described and then the kind of computational problems which may be solved will be described. Following that, the classic "traveling salesman problem" (TSP) will be discussed as a specific example of a more complex computation problem. The nature of this TSP is similar to the categorization problems inherent to perception, such as character recognition, but no appropriate simple model is available as a specific example of perception. The kinds of computational problems which lend themselves to solution by the present invention are those in which a rapid `good enough` discrete interpretation of input signals is desired, which can be described as a `good solution` (or a mathematical extremum) subject to conflicting constraints.
There are many problems outside of perception which are difficult because they involve the following aspects of perception.
1. The reduction of a set of analog input signals to a set of digital or quantized output signals which reflect the analysis of the objects and relationships in the scene. PA1 2. A single general Gestalt interpretation at a broad level is required. Because of the intrinsic noise and incompleteness in the sensory data, ambiguities of interpretation may remain at a fine scale. It is relatively unimportant from a perceptual viewpoint to disambiguate the fine details, and interpretations differing in fine details may be regarded as the same or equally valid. ("Best" is not definable with precision.) PA1 3. Invariance, or appropriate minor changes, in the interpretation over appropriate classes of change of input date, e.g., an object in the scene doesn't change by being moved one foot in the visual field, a background such as a lawn does not depend on locations of individual blades of grass, changing the angle of an object's elements such as a person's arms and legs does not change the person's relationship to another object in the scene . . . . PA1 4. As many as possible of the modes of processing and aspects of the data should be simultaneously brought to bear in parallel in searching for a "best" solution. The information is normally sufficiently strong that a time consuming (or hardware consuming) sequential consideration of special particular hypotheses is not necessary.
The placement of integrated circuit chips on a circuit board is typical of such problems. Given a circuit diagram which describes the interconnections between 75 chips, each having 16 pins, locate the chips on a circuit board such that, the total length of wire used is as small as possible, the length of the longest wire is as short as possible, the number of wires in different wiring channels is as uniform as possible, and the maximum wire depth is as small as possible. These criteria conflict, and a "best" arrangement involves trade-offs. The physical reasons for such criteria are such as to make them significant quantitative goals but not necessities. A really good solution tends to have many almost as good. There may also be many very good local optima which are comparably good, and the true object of a designer is to find one of these, not be obsessed with finding the absolute best solution. In any event the mathematical definition of "best" will be somewhat arbitrary weighing of rather different criteria. These characteristics are shared with perception problems.
The TSP is a classic example of such optimization. It is simple to describe, mathematically well characterized, and makes a good vehicle for describing the use of neuronal circuits to solve constrained optimization problems. A set of n cities, A, B, C, . . . have pairwise distance of separation d.sub.AB, d.sub.AC . . . , d.sub.BC . . . . The problem is to find a closed tour which visits each city once and has a minimum or short total path length. A tour defines a sequence 1, 2, 3, . . . m in which the cities are visited. For example, for five cities A, B, C, D, E, a typical tour might be to first visit B, then D, then A, then E, then C, then return to B. The total path length d of this tour would be EQU d=d.sub.BD +d.sub.DA +d.sub.AE +d.sub.EC +d.sub.CB
Like the problem of placing integrated chips on a board so as to minimize total interconnected path lengths, the selection of some particular short path segments precludes using some other short segments in making a closed tour.
The actual best TSP solution is computationally very hard to achieve because the problem is np (nonpolynomial) complete. Usually the practical problem does not require an exact solution; a good solution is acceptable. The practical problems of perception (or circuit wiring) are not concerned with exact solutions either, but rather in finding very good solutions. Most of the very good solutions have much in common, so a collective neural network constructed in accordance with the invention can be used to find a good solution. In easy perception problems, i.e., when identification of objects perceived is easy to make, the good solution found may be best. Similarly, in the TSP solution there may be arrangements of cities in which best is so unambiguous that the good solution found will in fact be best. The present invention is able to compute such a problem.
Collective neuronal circuits of the kind to be described can also perform the calculation required for a content addressable memory of the type that, given enough `clues` as input signals, will produce a very good response in the form of information that best matches the clues. As in other useful computations, the network of interconnected neurons involve the change of state of the network with time. The location of a particular stable point in state space is the information of the particular memory. The information in memory is reached not by knowing an address, but rather by supplying in the initial state any subpart of the memory of adequate size, i.e., enough `clues`.